Approaches and informatics tools to assist in the integration of similar clinical research questionnaires.

OBJECTIVE: The integration of similar clinical research questionnaires is a complex process that can benefit from informatics approaches and tools that provide a systematic structure for performing mapping and integration. This systematic approach is necessary to address complex issues in integration such as data heterogeneity, differing levels of granularity of questions and responses, and other issues involving semantic differences. Informatics tools and approaches have been successfully applied to various standard clinical vocabulary integration processes but not for questionnaire integration or mapping. METHODS: A systematic approach to questionnaire integration was developed in the context of a collaboration of researchers using Trial/DB, a database designed to support clinical research. This approach was applied to the integration of questionnaires involving breast cancer risk factors from each of three research sites. RESULTS: From 375 questions on the three original questionnaires, we identified 65 concepts that were measured by two or three of the sites. An algorithm was developed and used to formalize the process of mapping questions and answers across the questionnaires. The approach was applied to previously collected data and prospective data in disparate data-base systems to import and merge the data from these three sites into Trial/DB. CONCLUSION: Informatics tools that support a systematic approach to mapping questionnaires can be used throughout the research process from questionnaire integration and creation, legacy data integration to data library maintenance and curation.

Issues in computer-based decision support in public health illustrated using projects involving childhood immunization.

The article discusses issues that arise in implementing computer-based decision support using childhood immunization as a source of examples. The examples include (1) IMM/Serve, a program that provides patient-specific recommendations, (2) informatics tools developed to help validate IMM/Serve's knowledge and functionality, and (3) tools developed to explore how computers can help ensure immunization data quality. Issues discussed include the complexity of creating computer-based decision support, the need for a continuing process of revision and testing as the health field evolves over time, and the potential value of computer-based tools to assist in this process.

Maintaining and incrementally revalidating a computer-based clinical guideline: a case study.

The paper explores the issues involved in maintaining the logic within a complex computer-based clinical guideline, using as a case study IMM/Serve, an operational guideline whose domain is childhood immunization. For a period of more than a year and a half, we have maintained a log of (1) the national changes to the immunization recommendations, (2) the local customizations of IMM/Serve's logic, and (3) certain logic problems that arose in the process of accommodating these changes and customizations. We describe the nature of these changes, customizations, and problems. We also discuss how different types of domain knowledge might assist in the automated process of validating successive versions of the logic. The paper's goal is to use the immunization domain to provide specific examples of the issues and problems that arise in maintaining a computer-based clinical guideline.

Exploring the utility of demographic data and vaccination history data in the deduplication of immunization registry patient records.

Duplicate patient records pose a major problem for many immunization registries, as well as for many electronic patient record systems. This paper reports two complementary studies exploring the deduplication of immunization registry records. One study explores the utility of different demographic data elements, singly and in combination, to assist in the deduplication process. The second study explores how clinical patient data (vaccination history data) might assist in this process. To assess the utility of demographic data elements, data were used from three registries after duplicates had been identified. A computer program, IMM/Scan, was written to count the number of true-positive (TP) matches and false-positive (FP) matches found when using different Boolean combinations of demographic data elements. In this study, a strategy of "ORing high value ANDed pairs of data elements" appeared to be most powerful. To assess the utility of vaccination history data, record pairs were drawn from 440,000 patient records. Two metrics on patient history were tested: (1) the number of identical doses shared by two records, and (2) the number of "extra" doses in the combined history of two records. In this study, sample findings include: (1) for pairs of nonduplicate records, 93% had no identical doses and 90.6% had "extra" doses, and (2) for pairs of duplicate records, 83.8% had one or more identical doses and 82% contained no "extra" doses. These studies demonstrate potentially useful approaches to using demographic data and patient history data to assist the automated deduplication of immunization patient records.

Informatics issues in the national dissemination of a computer-based clinical guideline: a case study in childhood immunization.

IMM/Serve is a computer-based guideline that provides patient-specific recommendations regarding childhood immunization. IMM/Serve is currently installed at an estimated 75 sites within the US Indian Health Service (IHS), with plans to extend its use to roughly 150 additional sites nationwide by the end of the year 2000. The dissemination of IMM/Serve within the IHS provides a case study with concrete examples that illustrate the diverse informatics issues that arise in the widespread dissemination of a computer-based clinical guideline.

IMM/Scrub: a domain-specific tool for the deduplication of vaccination history records in childhood immunization registries.

IMM/Scrub is a pilot tool developed to assist in the deduplication of vaccination history records in childhood immunization registries. This problem is complicated by a number of factors including that fact that: (1) some doses are numbered and some are not, (2) doses may have different dose numbers, (3) doses may specify different preparations within a vaccine series, (4) one dose may indicate a combination vaccine and the other dose may specify one component of that combination, (5) two doses may have slightly different dates, and (6) combinations of any of these problems may occur together. IMM/Scrub is designed to help detect 10 different types of vaccination dose duplicates and also allows the user to specify flexibly the conditions in which a duplicate dose might be automatically eliminated. In addition, IMM/Scrub is linked to the IMM/Serve immunization forecasting program, which can provide additional assistance in the data cleaning process. The paper describes (1) the design of the current pilot implementation of IMM/Scrub, (2) the lessons learned during its implementation, and (3) our preliminary experience applying it to data from three immunization databases, from a state, a metropolitan area, and an academic medical center.

Exploring three approaches for handling incomplete patient histories in a computer-based guideline for childhood immunization.

A significant problem faced by immunization registries is that the dates of a patient's previous vaccinations may not be known. These incomplete histories can pose a problem when attempting to use a computer-based guideline to produce patient-specific immunization recommendations automatically. This paper describes an overall approach, together with 3 specific strategies, developed to help deal with this problem. The paper then describes our experience applying the approach to a database containing over 400,000 immunization histories. The paper also discusses a number of the issues raised in adapting a computer-based guideline to accommodate incomplete patient data of this sort.

Using semantic constraints to help verify the completeness of a computer-based clinical guideline for childhood immunization.

The paper describes Commander, a prototype computer program designed to help verify the completeness of a computer-based clinical practice guideline built using if then rules. It also describes the application of Commander to a guideline for childhood immunization. Commander is designed to help identify incomplete rule sets, where there are clinically meaningful conditions to which the guideline does not respond. To allow this, the user defines semantic constraints, in the form of if-then statements, which indicate combinations of conditions which are not meaningful. In an iterative process, Commander takes the guideline rules, together with an increasingly refined set of constraints and helps focus in on any combinations of conditions to which the guideline does not respond. When applied to the clinical guideline for childhood immunization, Commander was able to dramatically reduce the number of potential combinations of conditions for consideration and also identified several areas of incompleteness in the rules.

Tools for immunization guideline knowledge maintenance. II. Automated Web-based generation of user-customized test cases.

IMM/Test is a prototype software tool built to generate test cases that can be used to help test and verify the internal logic of an immunization forecasting program. A forecasting program takes as input a child's immunization history and produces recommendations as to which vaccinations are due and which should be scheduled next. IMM/Test was developed to test a specific immunization forecasting program, IMM/Serve. In addition, IMM/Test has been incorporated into a broader Web-based tool, IMM/Web, which allows the user (e.g., a member of an immunization registry staff) to customize the parameters used for immunization forecasting (e.g., the minimum ages for each dose and the minimum wait intervals between doses) to reflect local practice. IMM/Web then generates a customized set of test cases that may be used to test the user's immunization forecasting program. The user may also request that the test cases be automatically passed to IMM/Serve to analyze using the newly defined parameters. The paper describes the internal design of IMM/Test and IMM/Web and discusses certain lessons learned in the implementation of the two programs.

Issues in accommodating national changes and local variation in a computer-based guideline for childhood immunization and in related knowledge maintenance tools.

As clinical practice guidelines are increasingly implemented in computer-based form, a major challenge will be to maintain their domain knowledge as new national recommendations are developed and as local customization is required. This maintenance may also need to be performed for any computer-based tools developed to help in the guideline knowledge maintenance process itself. This paper uses the domain of childhood immunization to explore certain issues involved. It describes 1) two recent changes to the national recommendations dealing with the DTP and Polio vaccine series, and 2) several customizations requested by the immunization registries of the State of Oregon and the US Indian Health Service. It then describes how these guideline practice variations are currently handled in three computer-based tools, IMM/Serve, IMM/Def, and IMM/Test. Finally, it discusses how the use of these tools can provide one approach to characterizing the complexity of guideline variations.

Visualizing the logic of a clinical guideline: a case study in childhood immunization.

IMM/Graph is a visual model designed to help knowledge-base developers understand and refine the guideline logic for childhood immunization. The IMM/Graph model is domain-specific and was developed to help build a knowledge-based system that makes patient-specific immunization recommendations. A "visual vocabulary" models issues specific to the immunization domain, such as (1) the age a child is first eligible for each vaccination dose, (2) recommended, "past due" and maximum ages, (3) minimum waiting periods between doses, (4) the vaccine brand or preparation to be given, and (5) the various factors affecting the time course of vaccination. Several lessons learned in the course of developing IMM/Graph include the following: (1) The intended use of the model may influence the choice of visual presentation; (2) There is a potentially interesting interplay between the use of visual and textual information in creating the visual model; (3) Visualization may help a development team better understand a complex clinical guideline and may also help highlight areas of incompleteness.

Combining tabular, rule-based, and procedural knowledge in computer-based guidelines for childhood immunization.

IMM/Serve is a computer program which implements the clinical guidelines for childhood immunization. IMM/Serve accepts as input a child's immunization history. It then indicates which vaccinations are due and which vaccinations should be scheduled next. The clinical guidelines for immunization are quite complex and are modified quite frequently. As a result, it is important that IMM/Serve's knowledge be represented in a format that facilitates the maintenance of that knowledge as the field evolves over time. To achieve this goal, IMM/Serve uses four representations for different parts of its knowledge base: (1) Immunization forecasting parameters that specify the minimum ages and wait-intervals for each dose are stored in tabular form. (2) The clinical logic that determines which set of forecasting parameters applies for a particular patient in each vaccine series is represented using if-then rules. (3) The temporal logic that combines dates, ages, and intervals to calculate recommended dates, is expressed procedurally. (4) The screening logic that checks each previous dose for validity is performed using a decision table that combines minimum ages and wait intervals with a small amount of clinical logic. A knowledge maintenance tool, IMM/Def, has been developed to help maintain the rule-based logic. The paper describes the design of IMM/Serve and the rationale and role of the different forms of knowledge used.

Evaluating IAIMS at Yale: information access.

OBJECTIVE: To evaluate use of information resources during the first year of IAIMS implementation at the Yale-New Haven Medical Center. The evaluation asked: (1) Which information resources are being used? (2) Who uses information resources? (3) Where are information resources used? (4) Are multiple sources of information being integrated? DESIGN: Measures included monthly usage data for resources delivered network-wide, in the Medical Library, and in the Hospital; online surveys of library workstation users; an annual survey of a random, stratified sample of Medical Center faculty, postdoctoral trainees, students, nurses, residents, and managerial and professional staff; and user comments. RESULTS: Eighty-three percent of the Medical Center community use networked information resources, and use of resources is increasing. Both status (faculty, student, nurse, etc.) and mission (teaching, research, patient care) affect use of individual resources. Eighty-eight percent of people use computers in more than one location, and increases in usage of traditional library resources such as MEDLINE are due to increased access from outside the Library. Both survey and usage data suggest that people are using multiple resources during the same information seeking session. CONCLUSIONS: Almost all of the Medical Center community is using networked information resources in more settings. It is necessary to support increased demand for information access from remote locations and to specific populations, such as nurses. People are integrating information from multiple sources, but true integration within information systems is just beginning. Other institutions are advised to incorporate pragmatic evaluation into their IAIMS activities and to share evaluation results with decision-makers.

A prototype Web site for immunization knowledge maintenance.

IMM/Web is a prototype Web site designed to assist in the maintenance of the knowledge required to perform computer-based forecasting for childhood immunization. IMM/Web operates in conjunction with IMM/Serve, a immunization forecasting program that takes a child's immunization history and produces recommendations as to which vaccinations are due, and which should be scheduled next. IMM/Serve's domain knowledge is expressed in both tabular and rule-based form. Using IMM/Web, the various tabular forecasting parameters can be modified via the Web. Test cases can then be generated automatically which can be used to help verify the new version of logic. Finally, the test cases can be automatically passed for IMM/Serve to analyze using the newly defined parameters. The IMM/Web project is exploring how the process of updating, customizing, and testing new versions of a computer-based clinical guideline might be performed and guided in an organized fashion via the World-Wide Web.

IMM/Serve: a rule-based program for childhood immunization.

A rule-based program, IMM/Serve, is being developed to help guide childhood immunization for initial use, within Oregon. The program is designed primarily for automated use with an online immunization registry, but can also be used interactively by a single user. The paper describes IMM/Serve and discusses 1) the sources of complexity in immunization logic, 2) the potential advantages of a rule-based approach for representing that logic, and 3) the potential advantage of such a program evolving to become the standard of care. Related projects include 1) a computer-based tool to help verify the completeness of the logic, 2) a tool that allows a central part of the logic to be generated automatically, and 3) an approach that allows visualization of the logic graphically.

Medical Informatics Training at Yale University School of Medicine.

The Yale Center for Medical Informatics has offered Medical Informatics training since 1986. The paper describes the overall structure of the training program, focusing primarily on the postdoctoral fellowship, a major component of which involves an independent project which the fellow carries out under faculty supervision. The paper outlines a number of areas in which such projects have been performed, describes the previous backgrounds and post-training career directions of the trainees, and discusses certain observations based on our experience with the program.

Trade-offs in producing patient-specific recommendations from a computer-based clinical guideline: a case study.

This case study explored 1) how much online clinical data is required to obtain patient-specific recommendations from a computer-based clinical practice guideline, 2) whether the availability of increasing amounts of online clinical data might allow a higher specificity of those recommendations, and 3) whether that increased specificity is necessarily desirable. The "quick reference guide" version of the guideline for acute postoperative pain management in adults, developed by the Agency for Health Care Policy and Research, was analyzed. Patient-specific data items that might be used to tailor the computer's output for a particular case were grouped into rough categories depending on how likely they were to be available online and how readily they might be determined from online clinical data. The patient-specific recommendations were analyzed to determine to what degree the amount of text produced depended on the online availability of different categories of data. An examination of example recommendations, however, illustrated that high specificity may not always be desirable. The study provides a concrete illustration of how the richness of online clinical data can affect patient-specific recommendations, and describes a number of related design trade-offs in converting a clinical guideline into an interactive, computer-based form.

Lessons learned from a pilot implementation of the UMLS information sources map.

OBJECTIVE: To explore the software design issues involved in implementing an operational information sources map (ISM) knowledge base (KB) and system of navigational tools that can help medical users access network-based information sources relevant to a biomedical question. DESIGN: A pilot biomedical ISM KB and associated client-server software (ISM/Explorer) have been developed to help students, clinicians, researchers, and staff access network-based information sources, as part of the National Library of Medicine's (NLM) multi-institutional Unified Medical Language System (UMLS) project. The system allows the user to specify and constrain a search for a biomedical question of interest. The system then returns a list of sources matching the search. At this point the user may request 1) further information about a source, 2) that the list of sources be regrouped by different criteria to allow the user to get a better overall appreciation of the set of retrieved sources as a whole, or 3) automatic connection to a source. RESULTS: The pilot system operates in client-server mode and currently contains coded information for 121 sources. It is in routine use from approximately 40 workstations at the Yale School of Medicine. The lessons that have been learned are that: 1) it is important to make access to different versions of a source as seamless as possible, 2) achieving seamless, cross-platform access to heterogeneous sources is difficult, 3) significant differences exist between coding the subject content of an electronic information resource versus that of an article or a book, 4) customizing the ISM to multiple institutions entails significant complexities, and 5) there are many design trade-offs between specifying searches and viewing sets of retrieved sources that must be taken into consideration. CONCLUSION: An ISM KB and navigational tools have been constructed. In the process, much has been learned about the complexities of development and evaluation in this new environment, which are different from those for Gopher, wide area information servers (WAIS), World-Wide-Web (WWW), and MOSAIC resources.

Building a Database of Data Sets for Health Services Research.

The Database of Data Sets (DB/DS) for Health Services Research will be an online searchable directory of data sets which are available, often with restrictions and confidentiality safeguards, for use by health care researchers. The DB/DS project is aimed at a wide audience, and intends to include a very broad range of health care data sets, ranging from state hospital discharge data bases, to national registries and health survey data sets, to institutional clinical databases. The intended users are the same community of researchers, policy-makers, administrators and practitioners who are served by the National Library of Medicine's current bibliographic databases. This paper describes a pilot phase of the DB/DS project in which the issues involved in creating such a database were explored with an initial set of 20 representative data sets.